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Navigating the Problem Space of Academia: 

Exploring Processes of Course Design and Classroom Teaching 

in Postsecondary Settings 

Matthew T. Hora 

Postsecondary institutions in the United States are under increasing public and political 
pressure to demonstrate their delivery of a high-quality educational experience that leads to 
demonstrable outcomes including student learning and employability in the labor market (Arum 
& Roksa, 2011; Bok, 2006). Many critiques of higher education and current policy initiatives 
center on the instrumental role of classroom teaching, and how the widespread adoption of 
techniques grounded in the learning sciences could lead to such positive outcomes (e.g., 
President’s Council of Advisors on Science and Technology, 2012). As a result, a growing area 
of interest in both research and policymaking circles is whether faculty 1 effectively incorporate 
techniques such as problem-based learning (Hmelo-Silver, 2004) or Peer Instruction (Mazur, 
1997) into their teaching, and, if not, what contributes to the rejection of these techniques. 

However, most analytic attention is on what happens in the classroom and not the 
planning processes that take place long before a class begins. This is problematic because 
improved teaching and student learning is as dependent upon faculty developing a pedagogically 
informed and deliberate approach to curriculum planning as it is to sound classroom practice. 
Indeed, instructional refonns such as the scientific teaching approach (Handelsman et ah, 2004) 
encourage faculty to use formative and summative assessment data in an ongoing, iterative 
approach to planning and teaching their courses. Yet, as Joan Stark observed, “most attempts to 
improve teaching and learning in colleges have focused on the teacher’s role as a ‘classroom 
actor’ rather than as an ‘academic planner’” (Stark, 2000, p. 413). Thus, understanding the 
dynamics underlying faculty decisions about whether or not to adopt instructional innovations 
requires an expansive view of faculty practice that encompasses their roles as academic planners, 
not merely classroom actors. 

Furthermore, course planning and classroom teaching practices should be situated within 
their unique disciplinary and professional contexts. The ways in which the task environment 
influences how teachers approach classroom instruction (Shavelson & Stern, 1981; Schoenfeld, 
2000), curriculum design (Remillard, 2005), and policy implementation (Coburn, 2001) has long 
been an active and influential area of research in K-12 settings. Notably, these lines of inquiry 
are deeply influenced by theory and method from cognitive psychology and the learning 
sciences, especially in exploring the subtle dynamics whereby individuals perceive and interpret 


1 By faculty I mean all people who hold undergraduate teaching positions — whether full- or part-time, tenured or 
untenured — in postsecondary institutions with the exception of graduate teaching assistants. As such, an 
examination of the effects of different postsecondary appointments on pedagogical beliefs and/or instructional 
decision making is beyond the purview of this paper. For more details regarding the sample, see pp. 7ff. 
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their situations in ways that shape their decisions and actions (Greeno, 1998). The interactions 
between individuals’ attributes and their environments as they make decisions are a critical 
feature of educational practice, and thus a potentially important leverage point for facilitating or 
affecting change (Spillane, Reiser, & Reimer, 2002). 

In contrast, the literature exploring how postsecondary faculty plan and teach their 
courses tends to adopt a “black box” approach to the relationships among context, decision 
making, and practice. For example, a common way to conceptualize the influence of contextual 
factors on teaching is captured by a model used by Umbach (2007), which posits that “Faculty 
subcultures (professional, institutional, and disciplinary)— ^faculty teaching— > student learning” 

(p. 266). Besides assuming a linearity of cause-effect relations that may not in fact exist, such an 
approach leaves the specific and subtle dynamics between situations and teaching behaviors 
unexamined. Promising lines of inquiry shed light on these interactions in colleges and 
universities, including examinations of how pedagogical beliefs and goals are “filtered” by the 
local context (Stark, 2000), how faculty interpret and navigate their institutional and 
sociocultural environments (e.g., Trowler & Knight, 2000), and how “situational factors” such as 
class size are perceived by faculty as constraining or affording the adoption of pedagogical 
innovations (Henderson & Dancy, 2007). 

Yet, little empirical research has been conducted on the specific processes whereby 
faculty decisions about teaching are influenced by their environments, particularly from a 
practice-based perspective. Indeed, educational researchers across the K-16 spectrum have 
advocated for more practice-based research on teaching, so that how and why educators make 
decisions in “the wild” of classrooms and school hallways is better understood (e.g., Bastedo, 
2012; Halverson & Clifford, 2006). In this paper, I report findings from an exploratory study that 
focused on the instructional decisions made by a group of 58 postsecondary math and science 
faculty while planning their classes in three public research universities. At the heart of this 
investigation is the notion of a “problem space,” which is an idea from cognitive science that 
refers to how a task is perceived by the problem solver, and how subsequent goals and strategies 
flow from this initial construction of the situation (Newell & Simon, 1972). 

Using data from a freelisting exercise, retrospective recall interviews, and in-person 
observations of classroom teaching, I use the problem space framework to address the following 
questions: (1) What factors do faculty perceive as most salient to their course planning activities? 
(2) What specific strategies do faculty use when planning a specific course? (3) Which curricular 
artifacts, if any, are developed by faculty and how do they plan to enact them in the classroom? 
(4) How do these processes of course planning unfold in concrete instances? 

Answers to these questions promise to contribute to the literature on instructional 
decision making in both theoretical and practical ways. By rejecting the notion that a single 
variable, or “magic bullet,” can explain how and why faculty teach the way they do, this study 
brings theory and method from the cognitive and learning sciences to bear upon the important 
problem of delineating the combination of factors that collectively shape how faculty view their 
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work in real-world situations. Insights into these critical aspects of instructional decision making, 
which previously have remained an unexamined empirical problem, not only promise to 
contribute to the field of postsecondary education’s understanding of faculty work and the 
dynamics of educational refonn in postsecondary institutions (Spillane, Reimer, & Reiser, 2002), 
but also may have practical applications. In particular, evidence regarding faculty decision- 
making can be used to illuminate “leverage points” or organizational factors that educational 
leaders could support or otherwise alter in order to create the conditions for improved teaching 
and learning (Schoenfeld, 2000; Spillane, Halverson, & Diamond, 2002). 

Background 

Extensive bodies of research have explored the dynamics among instructional decision 
making, organizational contexts, and teaching practice in both higher education and K-12 
schools. Because a survey of these various literatures is not within the purview of this paper, in 
this section I provide a brief overview of the key literatures that have infonned and inspired the 
present investigation. These include research on curriculum design in both postsecondary 
institutions and K-12 settings, with a focus on studies of decision making in the classroom from 
a cognitive perspective. I then discuss how problem space theory, as well as conceptual tools 
from other theoretical traditions including situated cognition, can be used to interrogate the 
under-studied phenomenon of how faculty plan and then teach their courses. 

Postsecondary Research on Curriculum Design 

Much of the early work on course planning and curriculum design in higher education 
was largely conceptual, and, instead of examining how faculty went about designing their 
courses and programs, researchers sought to develop explanatory models of planning largely to 
aid designers in developing and improving degree programs (Conrad & Pratt, 1983; Dressel, 
1980; Mayhew & Ford, 1971). These efforts entailed complex models that included numerous 
boxes and arrows denoting causal relations among factors. For example, DresseTs (1980) model 
posited six continua upon which programs can be characterized (e.g., continuity to 
fragmentation) that administrators and curriculum designers should consider as they designed 
new programs. However, Conrad and Pratt (1983) critiqued this approach to modeling curricular 
design for ignoring the constituent parts of the planning process and the myriad contextual 
influences that shape them, as well as being an overly linear accounting of curriculum 
development that likely did not reflect how curricula were designed in real-world settings. 
Instead, their model comprises a recursive and non-linear process whereby a wide range of 
factors shape how curricular decision making unfolds, including environmental inputs (e.g., 
professions, community), the nature of the material, instructional traditions, and resource 
allocations, to name but a few (Conrad & Pratt, 1983). 

This shift from abstract models of curriculum design to more grounded, descriptive 
accounts of how faculty designed their courses in practice was the hallmark of the research 
program of Joan Stark (Stark & Fowther, 1986). Motivated in part by advances in cognitive 
science, Stark and colleagues sought to identify precisely how deliberate faculty were in 
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designing their courses. Given the autonomy that many postsecondary faculty enjoy in planning 
their courses, Stark hypothesized that personal characteristics such as beliefs about teaching and 
learning, and experiences as an instructor, would largely drive their decisions about the structure 
and content of a course. However, these individual-level factors did not unilaterally dictate 
faculty behaviors, as they were influenced or “filtered” by features of the organizational context 
that ultimately detennined how a course was designed and taught (Stark, 2000). A key finding 
from this research program was that much of postsecondary instructors’ planning for a 
previously offered course (versus planning for an entirely new offering) involves minor 
alterations to existing materials on account of the fact that instructors often teach the same 
course(s) year to year — what Stark (2000, p. 420) called “routine maintenance.” 

This body of evidence remains the most extensive and comprehensive work on faculty 
course planning to date, but, given changes in the nature of higher education and faculty teaching 
in the intervening 25 years, the field would benefit from more recent empirical work on these 
topics (see Lattuca & Stark, 2011 for a more prescriptive approach to curriculum design). 
Additionally, this work was primarily survey based and thus did not capture or describe the 
subtle features of faculty decision making as it unfolded in real-world situations. The importance 
of such descriptive work was noted by Stark (2000, p. 435) who stated, “Our work fell short of 
exploring in depth the actual decisions teachers make about course plans and curriculum.” 

While the notion of contextual filters suggests a theoretical grounding in work on 
perception and situated cognition, it was not until scholars in the United Kingdom and Australia 
began to investigate the nature of faculty thinking that research on instructional decision making 
took a decidedly cognitive turn. Much of this work focused on identifying particular conceptions 
or approaches to teaching that, in turn, were theorized to shape an instructor’s classroom practice 
(see Hativa & Goodyear, 2001 for a review). A particularly promising line of inquiry that 
explicitly draws on theory from cognitive science to examine these processes is that of McAlpine 
and colleagues (2006), who employed the problem space construct to examine how faculty 
develop internal mental models of their social and organizational environment. Of particular 
salience to the current policy environment focused on instructional reform is an analysis of how 
physics faculty perceived the “situational factors” that either impeded or supported their adoption 
of more interactive techniques, which found student attitudes towards school, organizational 
expectations of content, and the lack of instruction-oriented time to be influential (Henderson & 
Dancy, 2007). It is this line of inquiry that the current paper builds upon by developing a more 
theoretically robust framework for studying curriculum design and classroom teaching, one that 
is strongly influenced by the extensive research on teacher decision making in K-12 schools. 

Research on Curriculum Design and Enactment in K-12 Settings 

The focus on teacher thinking as an object of research was largely motivated by the view 
that, as Fullan stated, “educational change depends on what teachers do and think — it’s as simple 
and as complex as that” (2001, p. 1 15). A core idea that motivated early research on teacher 
cognition is that teachers are complex decision makers whose problem-solving capabilities are 
shaped by characteristics (and constraints) of human cognition as well as features of the 
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instructional situation (Lee & Porter, 1990). In tenns of curriculum design, researchers have 
pointed out that teachers’ adapt materials to fit their strengths and views as instructors, as well as 
the unique needs of their students and constraints of their classrooms (Shavelson & Stern, 1981). 
As a result, not a single “curriculum” designed by educational experts is directly implemented in 
the classroom; instead there exist three instantiations of the curriculum based on teachers 
adapting materials to fit their own views and situations: the formal curriculum, the planned 
curriculum, and the curriculum actually enacted (Gehrkey, Knapp, & Sirotnik, 1992). This view 
represented a departure for the field, which had previously conceptualized the curriculum as a 
“fixed” text that was directly followed (or not) in the classroom with more or less fidelity 
(Remillard, 2005). 

These distinctions were crucial for the field because they highlighted the active role 
teachers take in interpreting curricular materials to their classroom applications. This focus on 
interpretation and the central role of perception in shaping behavior has also been used to study 
policy implementation and education refonn, where the processes whereby teachers interpret, 
adapt, or reject innovations are the primary consideration (e.g., McLaughlin, 1987; Spillane, 
Reiser & Refiner, 2002). According to this view, the analytic focus should be on the teacher as 
he or she interprets and makes sense of the curricular text rather than on the formal curriculum 
and the teacher’s varying degree of fidelity to it. 

For example, Otte (1986) crucially distinguished between the curricular artifact as an 
“objective scheme” (i.e., the physical form of a text) and the “subjective scheme” of how these 
texts are perceived, interpreted, and integrated into a teacher’s practice. Remillard (2005, p. 255) 
adopted the view that teachers have a “participatory relationship” with the curriculum. But it is 
important to recognize that teachers do not interact with these texts in isolation, but instead they 
encounter them in specific cultural and organizational contexts “that assign to the curriculum a 
particular meaning” (Remillard, 2005, p. 234). This is evident in part through the curricular 
artifact itself, which represents an important instantiation of cultural norms and beliefs for a 
group (i.e., school district, discipline, institution), such that the materials may act as cultural tools 
that mediate as well as constrain instructional behavior in subtle ways (Brown, 2002). While 
research in this area foregrounds the critical role of perception in course design and classroom 
teaching, the processes whereby teachers’ subjective schemes are generated in specific settings 
and then influence practice are less well examined. 

Focus on Perception in Context: Problem Space Theory 

Thus, a critical problem facing the field is how to conceptualize the interactions between 
faculty perceptions of their contexts and subsequent decisions about curriculum design and 
classroom teaching. Research on problem solving from a situated cognition perspective provides 
robust conceptual tools to extend our understanding of these dynamics. A fundamental idea in 
teacher cognition research is that the rational choice model, which posited that humans solely 
make logical decisions based on a deliberate search for the best alternative among all possible 
choices, is not sufficient to explain how educators make decisions in practice (Lee & Porter, 
1990). Instead, in addition to some element of conscious deliberation, human decision making is 


5 



Navigating the Problem Space of Academia 


viewed as limited by certain principles of cognition (e.g., perception, memory capacity, 
limitations in information processing). As a result, this process of deliberation operates on both a 
conscious and subconscious level, and is largely shaped by the individual’s unique 
characteristics and the nature of the situation (Greeno, 1998). An important component of this 
perspective is that decision makers construct a simplified mental model, or cognitive 
representation, of the task. This model is known in cognitive science as a “problem space” and is 
shaped by the individual’s perception of the nature of the task, its goals, and the strategies that 
can best lead to the successful performance of the task (Newell & Simon, 1972). Constructing 
this simplified model of the task allows the problem-solver to manage the overwhelming 
perceptual inputs of the environment into a representation that is cognitively efficient (Lee & 
Porter, 1990). 

Because problem spaces can be large and complex, Newell and Simon (1972) theorized 
that rule-like operators that link perceptions of the task with specific strategies would be used as 
cognitive shortcuts or “heuristics” that are reinforced over time through their successful use in 
solving particular problems, and thus allow individuals to minimize cognitive load in real-world 
tasks (Goldstein & Gigerenzer, 2002). One type of heuristic, called a “perceived affordance,” 
refers to the relationship between perception and potential activities within a given environment. 
The notion of affordances was a key aspect of ecological perception theory, which focused on 
how the perception of particular objects involved both the reception of visual infonnation as well 
as the instantaneous interpretation of the actionable properties of these objects (Gibson, 1979). 
For example, an individual surveying a room may visually perceive a chair and certain properties 
related to how she could interact with it (e.g., sitting). Later researchers focused less on the entire 
range of potential actionable properties of objects and instead on those behaviors that individuals 
perceive as possible and desirable in a given situation (Nonnan, 1990). Thus, if the chair was 
located in a room full of professional wrestlers, perhaps the individual would perceive the chair 
not as an object for sitting but as a potential weapon. 

Similarly, researchers of teacher cognition emphasized how individual teachers perceived 
certain objects or situations in their schools or classrooms in conjunction with particular 
responses or instructional strategies (Leinhardt & Greeno, 1986). Over time educators will 
develop attunements or cognitive heuristics related to the constraints and affordances represented 
by objects, policies, behavioral norms, and social regularities in their schools and classrooms 
(Greeno, 1998; Lee & Porter, 1990). Some researchers pay particularly close attention to the role 
of designed objects or “artifacts” (such as curricular texts) in shaping how teachers make 
decisions, based on the idea that activity is mediated and transformed by the unique 
configuration of artifacts within a person’s environment (Wertsch, 1991; Halverson, 2003). 

A New Approach to Studying Instructional Decision Making in Postsecondary Settings 

So what do the concepts of problem spaces, cognitive heuristics, and perceived 
affordances have to do with the issue outlined at the outset of the paper, namely how 
postsecondary faculty plan and then teach their courses, and subsequent implications for 
educational improvement? It is widely recognized that teaching situations are substantially 
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influenced by contextual factors such as organizational policies or student background, and these 
contexts are often dynamic and evolving rather than fixed and stable (Borko, Roberts, & 
Shavelson, 2008). Recognizing the implications of this complexity for research on educational 
reform, Shavelson & Stem (1981, p. 461) argued more than 30 years ago that the field needs to 
understand teachers’ goals, the nature of their task environment, their cognitive capabilities, and 
the relationship among these elements. More recently, Greeno (1998) encouraged researchers to 
focus on the specific processes whereby teacher’s problem spaces are constructed and modified, 
and especially how perceived affordances play a role in shaping these processes. Yet, more than 
a decade later, little empirical research exists in this area and insights into the “actual decisions 
teachers make about course plans and curriculum” (Stark, 2000, p. 435) remain elusive. 

In this paper I address this gap in the literature by focusing on four steps of the decision 
making process: (1) situation recognition — where the context is filtered down to essential 
components, (2) strategies or action steps for accomplishing the task, (3) curricular artifacts 
created, and (4) planned enactment of artifacts in the classroom. (See Figure 1.) Then, to 
determine how decisions translate into classroom practice, I examine how the process unfolded 
for two individuals. 


Step I : Situation recognition Step 2: Matching situation Step 3: Creation of artifacts Step 4: Classroom 

and identification of goals to planning strategies and planned enactment practice 

in repertoire 

Select elements of the 
organizational context are 
perceived 



The individual’s pre-existing 
cognitive characteristics help to 
shape how (a) the task is 
conceptualized, and (b) how 
elements are perceived 
as affording (or constraining) 
certain strategies 


The task situation 
“activates” a limited 
number of strategies 
- these are retrieved 
from individual’s 
toolkit of 
experiences 


Figure 1. Model for construction of problem spaces during course planning 


This model is a starting point for elaborating the dynamics among individual decision making, 
the organizational and broader sociocultural contexts, and actual teaching practice. In this early 
stage of exploring these complex interactions I necessarily restrict the scope of the analysis to 
focus on the individual faculty and his or her teaching decisions. As such, this paper largely 
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ignores the critical influences of social dynamics, politics, and cultural narratives that comprise 
the academic field (Bourdieu, 1988; Trowler & Knight, 2000), the inclusion of which will be 
important for future research in this area. 


Methods 

This study is a descriptive analysis of faculty practice in naturalistic settings. Practice- 
based research is increasingly common in educational research, particularly in regard to 
phenomenon where little is known about the salient variables at work and the dynamics between 
context and behavior (Bastedo, 2012; Halverson & Clifford, 2006). Such descriptive analyses of 
social action that are grounded in individuals’ experiences and actions can shed light on subtle 
aspects of behavior that may generate new hypotheses, theory, and scientific insights (Flyvbjerg, 
2006). This qualitative case study is uniquely suited to producing such rich, detailed accounts of 
practice. The cases analyzed are of the course planning and classroom teaching practices of 
faculty in math, biology, chemistry, geology, and physics departments at three large, public 
research universities. The method involves an intensive analysis of a single bounded unit (Yin, 
2008). 


The three study institutions were selected on the basis of the interests of the larger study 
from which this analysis is drawn — that of educational practice in science and math disciplines at 
the undergraduate level. The three institutions share similar undergraduate populations 
(approximately 25,000), education-research activities in science and math, and number of 
science and math departments. Science and math disciplines were selected for this analysis due 
to the nature of the larger study of which it was a part. This study was supported by the National 
Science Foundation and focused on the underlying influences of faculty teaching in these 
disciplines. While each of the institutions had active pedagogical reform initiatives underway at 
the time of data collection, one (Institution C) had a particularly active cross-departmental 
program to encourage interactive teaching among faculty. It is possible that these initiatives 
influenced the nature of faculty perceptions and teaching practices. The sampling frame for this 
study included 170 individuals listed in the Spring 2012 timetable as the instructor of record for 
undergraduate courses in math, physics, chemistry, biology, and geology departments at the 
study sites. Where multiple departments existed for one of these disciplinary groups, these were 
collapsed together for the purposes of this study. Individuals were contacted up to two times via 
email for participation in the study, and 58 faculty ultimately self-selected into the study. 
Information about the study sample is provided in Table 1. 

It is important to note that data from the entire study sample (N=5S) were not used to 
address each of the research questions. For three respondents, answers to the freelist questions 
were insufficiently detailed to include in the analysis. For 1 1 respondents, answers to questions 
about course planning were not provided or were insufficiently detailed. The removal of these 
transcripts from the dataset thereby alters the composition of the study sample for the different 
analyses conducted in this paper. The three respondents removed from the freelist dataset were 
not included in the group removed from the planning dataset. Finally, for the in-depth case 
analyses I examined the interview and classroom observation data from a subsample of two 
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respondents who exhibited similarities in discipline, appointment type, and class size and course 
level. Given the considerable influence of factors such as course content and class size on 
planning processes (Stark, 2000), I selected respondents for the in-depth analyses who shared 
these characteristics in order to minimize the potential sources of variability in their behaviors. 


Table 1. Description of sample by institution (interviewees) 



Total 

Institution A 

Institution B 

Institution C 

Total 

58 

20 

18 

20 

Sex 

Female 

25 

11 

5 

7 

Male 

23 

9 

13 

13 

Discipline 

Math 

14 

4 

4 

6 

Physics 

9 

3 

4 

2 

Chemistry 

9 

4 

3 

2 

Biology 

14 

4 

5 

5 

Earth/space science 

12 

5 

2 

5 

Level of course 

Lower division 

36 

10 

11 

15 

Upper division 

22 

10 

7 

5 

Size of course 

50 or fewer 

22 

7 

6 

9 

51-100 

8 

1 

3 

4 

101-200 

21 

11 

4 

6 

20 1 or more 

7 

1 

5 

1 

Position type 

Lecturer (non-tenure track) 

25 

11 

4 

10 

Assistant professor 

9 

4 

3 

2 

Associate professor 

3 

3 

0 

0 

Professor 

21 

2 

11 

8 


Data Collection 

The data collected for this study include semi-structured interviews and classroom 
observations. Interview data were analyzed for the entire study sample of 58 faculty, whereas 
observation data were used only for the two participants in the in-depth analysis. All data were 
collected by three analysts who underwent training in the research protocols prior to fieldwork. 

Semi-structured interviews. The interview protocols comprised two components: a 
freelist exercise and a series of questions about the teacher’s decision-making processes. First, 
the freelist exercise was intended to elicit those factors that respondents consciously considered 
the most influential in shaping course planning. The freelist technique is commonly used in 
cognitive psychology and cultural anthropology research, especially to identify “emic,” or 
insider, cultural domains in ethnographic fieldwork (Bernard, 2011). This contrasts to an “etic,” 
or outsider, account, which could describe much of the prior work in this area. The method 
assumes that when people report terms they do so in order of familiarity and cognitive salience, 
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and that terms will most commonly reflect locally important items (Romney & D ’Andrade, 

1964). Researchers recommend that freelist prompts focus on a single mental category or domain 
in order to delimit potential responses (Quinlan, 2005), and in the data gathering portion of this 
study the three researchers focused on course planning for undergraduate classes. The prompt for 
the freelist was: 

Please list all of the things that come to mind that influence you as you plan a specific 
lecture or class at the undergraduate level, using only single words or phrases. For 
example, the things that influence how a tomato grows could be sunlight, good soil, 
fertile seeds, and so on. 

Some respondents answered the freelist question in complete sentences, whereupon the 
interviewer reminded them to provide only single words or short phrases. 

Second, we asked the respondents questions about how they planned for a specific class 
using a variation of the critical decision-making technique (Crandall, Klein, & Hoffman, 2006; 
Klein, 2008), which essentially is a retrospective think-aloud technique to elicit details about 
how a decision was made in a specific situation. An important feature of this approach is the use 
of in-depth probes (e.g., specific decision points, cues noticed during decision making) to 
examine subtle features of decision making that may not be reported by respondents during a 
concurrent think aloud (Feldon, 2010). For this study respondents were asked to report their 
specific steps while planning for a class that our research team would observe that week. Follow- 
up probes to respondents’ answers included questions about any situational cues that influenced 
the decision, any curricular artifacts that resulted from the planning process, and if and how these 
artifacts would be used in their actual classroom teaching. Interviews lasted 30-45 minutes and 
were digitally recorded. 

Classroom observations. The Teaching Dimensions Observation Protocol (TDOP) is a 
classroom observation instrument developed to produce fine-grained descriptions of instructional 
practice (See Hora & Ferrare, 2013). The version of the TDOP used in this study captured five 
different dimensions of teaching practice: teaching methods, pedagogical strategies, student- 
teacher interactions, cognitive engagement, and use of instructional technology. Within each 
dimension there exist several detailed codes that observers capture at 2-minute intervals in real 
time. Prior to gathering data in the field, the three researchers established a common 
understanding of each code through rigorous training that included in-depth discussions about 
the meaning of each code category and individual codes, practice coding of videoed class 
segments, and, finally, the testing of inter-rater reliability. 

Data Analysis 

Analytic techniques included saliency analysis of freelist data, thematic analysis of 
interview data, and thematic network analysis of the sequence and structure of decision-making 
processes for two respondents. Analyses described in this section were conducted by the author 
and another member of the research group. 
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Saliency analysis of freelist data. Freelist data can be analyzed in a variety of ways, 
including how frequently terms were mentioned, the position of terms, or the salience of terms 
that accounts for both frequency and position. For this analysis, we calculated the salience 
measure and used it as the primary indicator for situational recognition by respondents. Before 
conducting the analysis, it was necessary to prepare the data. This involved reviewing interview 
transcripts to ensure that respondents provided useable data for the freelist (i.e., single words or 
short phrases). Three respondents failed to discuss influential factors at all in their interviews and 
thus their data were not included in the analysis. In addition, 23 respondents provided 
information on these topics in sentence form, and not single words or short phrases. In these 
cases, it was necessary to identify passages where respondents answered the freelist prompt and 
to distill their more expansive observations into single words. To do this, two members of the 
research group met and engaged in a process of data reduction, which entailed an open-coding 
process whereby new codes or terms were derived from the data (Bernard, 2011; Miles & 
Huberman, 1994). Ideally, freelist data are provided solely in the form of single words or short 
phrases, but in this case an additional analytic step was necessary. 

Then, because respondents listed tenns that could be considered closely related but were 
in fact differently phrased (e.g., course material, content, and course content), a process of 
standardizing the terms was necessary. For this step, two of the analysts (including the author) 
reviewed the raw data and developed a list of terms that captured the intent of the entirety of 
terms provided. This involved making analytic decisions regarding the collapsing of multiple 
terms into a new tenn (e.g., course content). After we developed a final list, comprising 75 terms, 
we reviewed each respondent’s list of terms to ensure consistency with the master list. 


The finalized term lists were then organized according to the following groupings: the 
entire sample, by institution, and by discipline. We analyzed these lists using Anthropac 
(Borgatti, 1996), with the primary output measure being tenn salience for each of the groups. 
Salience is a commonly used metric because it suggests that a tenn reflects a shared cultural 
domain, or information that is psychologically important and relevant for multiple members of a 
group (Romney & D’Andrade, 1964). A salience measure was developed by Smith (1993) to 
reflect a mean percentile rank for each tenn across all respondent lists, that indicated the degree 
to which a term was both frequently cited and the order in which it was reported. For each 
individual list of terms, the salience measure is computed as: 


ZiL-Rj + 1 ) 

5 = r -)/N 


Where S is the average rank of a given term across all of the freelists in the study sample, 
weighted by the lengths of the freelists in which the term is found. In the fonnula, L=length of a 
list (e.g., number of items in a list); Rj = position of item j in the list of tenns (first item is 1), and 
n = number of lists in the sample. The resulting score is the average percentile rank of a 
particular term across all of the respondent lists — the item’s gross mean percentile rank (see also 
Smith & Borgatti, 1998). 


11 



Navigating the Problem Space of Academia 


Thematic analysis of interview data. Next, responses to questions about instructional 
planning were analyzed for all 58 respondents using an inductive approach to qualitative data 
analysis (Bernard, 2011; Miles & Huberman, 1994). The goals for this analysis were to discern 
the specific planning steps that respondents reported for their upcoming class, which curricular 
artifacts were produced (if any) as a result, and how these artifacts would be used in the 
classroom. To identify the planning steps across respondents, two of the analysts (including the 
author) independently reviewed a sample of ten transcripts and each developed a preliminary 
code list using an open-coding procedure. In reviewing the transcripts, we focused on sections 
where respondents answered the question about their planning. Below is an example of raw data 
reviewed during this analysis: 

Before each day I go through and read through my notes from last summer and make sure 
I am fresh on the topic and I will [identify] another example problem or two that I have 
not used before, because some kids are repeating the class. 

In creating the codes we attempted to capture higher-order categories related to planning 
actions, while also maintaining a close relationship to the fine-grained nature of the data. In this 
case, we identified a preliminary code of “reviewing old notes.” We then compared results and 
produced a revised code list, which was used to review another ten transcripts. At this stage of 
the analysis each successive instance of the code was compared to previous instances in order to 
confirm or alter the code and its definition (i.e., the constant comparative method) (Glaser & 
Strauss, 1967). We then assessed inter-rater reliability by calculating the percentage of 
agreement between the two analysts in applying the codes. The proportion of instances in which 
we both coded the same code relative to all instances was 94%. After another process of revising 
the codes, we compiled a final code list, whereupon I then reviewed the entire dataset using these 
codes. Reliability was ensured throughout the analytic process through regular meetings between 
myself and the second analyst, testing inter-rater reliability at different stages of the analysis, and 
careful documentation of the chain of decisions made that led to the final results reported in this 
paper. Using this same procedure we reviewed the transcripts to identify curricular artifacts and 
whether or not these artifacts were to be used in classroom teaching. 

Thematic network analysis of structure and sequence of decision making. Finally, 
the interview transcripts and classroom observation data for the two individuals selected for the 
in-depth analysis were analyzed using the thematic network analysis technique. This is a 
structured approach for identifying relationships between concepts in a graphic and time-ordered 
fashion (Miles & Huberman, 1994), and is similar to the verbal analysis method of Chi (1997). 
The core of this analysis for this paper was the identification of the distinct steps that comprised 
the individuals planning strategies or actions. To identify these related steps the interview 
transcripts were analyzed to identify explicit statements regarding relationships among different 
planning steps. These relationships were identified through two criteria: (1) respondents’ 
statements that indicated a relationship among factors, and (2) analyst interpretation of 
associations among factors. For example, one of the faculty included in this analysis clearly 
stated that when planning for a class she first retrieved her PowerPoint slides, then consulted the 
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news and research literature to identify any topics that should be included in the slides, and then 
made changes accordingly. These results were used to develop a graphic that depicted the 
planning steps that were reported as a class was being planned and taught. 

Then, we created another graphic for each individual that included each of the datasets 
included in the analysis: (1) freelist results, (2) planning steps, (3) curricular artifacts and 
planned use, and (4) classroom observation data. We analyzed the classroom observation data for 
each of the instructors by calculating the proportions that a particular code was observed in 
relation to all possible 5-minute intervals. Taken together, the graphics show what I call the 
“decision chains” that represent the structure and temporal nature of instructional decision 
making at the individual level. At this important stage, it was clear that both respondent’s 
reported a similar decision step, that of note retrieval and fine tuning, and so the two visuals were 
linked. It is important to note that the resulting displays represent the accounts of only two 
respondents from the study and should not be extrapolated to entire departments or institutions 
within the study sample or viewed as definitive accounts of action and behavior within these 
administrative units. 


Results 

In this section I report results for the different components of problem space construction 
for the study sample: situation recognition, actual planning steps, form of the curricular artifact 
and reported use of artifacts in the classroom. Then, for two individuals, I report how these 
processes unfolded in the context of a specific class, along with how they actually taught the 
class. 

1. Situation Recognition: Factors Involved in the Initial Stages of Problem Space 
Construction 

Results from the freelist exercise capture the specific features of the task environment 
noticed or perceived as influential to planning processes by the individual. It is important to note 
that the types of environmental factors viewed as salient were left entirely up to the individual, 
and responses included a wide range of factors such as organizational, sociocultural, and 
curriculum-related factors. Results are depicted in Table 2, showing the frequency with which 
were reported across all respondent’s lists, the average ra nk of the term across all freelists, and 
salience scores that measures both frequency and average rank. In the interest of space, only 
terms with salience scores over .10 are reported here. 

Across the entire sample, faculty perceive time (.25) as the most salient organizational 
factor influencing their planning, followed by class size (.20) and course content (.20). These 
factors are notable by being relatively “fixed” or durable features of the environment that are 
established by departmental administrators and/or curriculum committees. The next most salient 
tenn was the consideration of which illustrations or examples to use in teaching from either 
memory or textbooks (.13), and a curricular concern about the course textbook itself, particularly 
in regard to how material is sequenced and arranged (. 12). These data indicate that when faculty 


13 



Navigating the Problem Space of Academia 


consider the course design task — that is, the first stage of problem space construction — they first 
think about these factors. 


Table 2. Freelist results by institution 



Frequency 

(%) 

Average 

Rank 

Salience 

Total sample (N=55) 

Time 

26.3 

1.33 

0.25 

Class size 

26.3 

2.93 

0.20 

Course content 

24.6 

2.50 

0.20 

Examples/illustrations 

22.8 

4.31 

0.13 

Textbook 

15.8 

3.33 

0.12 

Institution A (n -20) 

Time 

36.4 

1.00 

0.36 

Class size 

40.9 

3.00 

0.30 

Examples/illustrations 

22.7 

3.40 

0.15 

Course content 

18.2 

2.50 

0.14 

Student level 

18.2 

3.00 

0.13 

Student background 

18.2 

4.25 

0.11 

Institution B (n -16) 

Examples/illustrations 

37.5 

4.50 

0.20 

Student career/degree trajectory 

25.0 

2.75 

0.17 

Syllabus 

25.0 

4.25 

0.17 

Textbook 

18.8 

2.33 

0.17 

Course content 

18.8 

2.67 

0.16 

Institution C (n —19) 

Time 

36.8 

1.71 

0.34 

Course content 

36.8 

2.43 

0.29 

Class size 

26.3 

3.20 

0.20 

Making it interesting 

26.3 

4.20 

0.14 

Textbook 

21.1 

4.50 

0.14 

Pedagogical goals 

21.1 

4.25 

0.12 

Course level 

15.8 

3.33 

0.10 


Note: Items with more than 10% salience scores are included in the table. 


At Institution A, time (.36), class size (.30), examples and illustrations (.15), and course 
content (. 14) were reported, all of which mirror the results from the entire study sample. 
However, two student-related factors were also reported: student level (.13) and student 
background (.11), which indicates that considerations about students enter into design 
considerations for this group of faculty at Institution A. At Institution B, illustrations and 
examples was the most salient term (.20), followed by student career trajectory (.17), syllabi 
(.17), textbooks (.17), and course content (.16). These data from Institution B are notable in the 
prominent roles that considerations of teaching techniques and students play in the problem 
space construction process. At Institution C, time (.34) was the most salient factor, followed by 
course content (.29), class size (.20), the pedagogical consideration of making the material 
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interesting (.14), the textbook (.14), and course level (.10). In this case, the fixed constraints of 
time, content, and class size constitute the primary considerations for faculty at Institution C. 

In order to visualize these results within the theoretical framework advanced in this 
paper, results from the freelist analysis for the entire sample and each institutional group are 
depicted in Figure 2, which visualizes the first stage of the problem space construction model. 



Entire Study Sample (n = 55) Institution A (n = 20) Institution B (n= I 6) Institution C (n= I 9) 

5 terms above .10 7 terms above .10 5 terms above .10 7 terms above . 1 0 

Figure 2. Stage 1: Situation recognition, factors involved in initial stages of problem space construction 

An interesting result is the institutional difference in tenns of salience. While it is 
impossible to detennine the precise source of this variability given limitations associated with 
the sample as well as the large number of potentially influential variables, one difference among 
the institutions that came across in the interviews was the active and highly visible pedagogical 
reforms initiative underway at Institution C. However, one could assume that such reforms may 
make faculty more cognizant of student and/or pedagogy-related factors. Yet, the institution 
where the fixed constraints are least salient (Institution B) is one where pedagogical reforms are 
less prominent than at the other two study sites. Thus, the source of the institutional differences 
cannot be addressed in this paper but remain an interesting problem to examine in future 
research. 

During analysis, similarities and differences among the reported factors became apparent, 
so we conducted an inductive thematic analysis to ascertain if individual factors could be 
categorized into higher-order themes. The analysis revealed that individual items could be 
categorized into five groups: (1) organizational factors (e.g., time, class size), (2) curricular 
factors (e.g., content/topic, textbooks), (3) pedagogical considerations (e.g., pedagogical goals, 
making it interesting), (4) teaching techniques (e.g., examples/illustrations, assessment), and (5) 
student factors (e.g., student career trajectory). This structure was used in the remainder of the 
analysis. 

2. Planning Steps: Specific Steps Taken by Faculty when Planning Courses 

After the parameters of the problem space are identified, a limited number of actual 
planning steps are then “activated” for instructors. In conducting the analysis it became apparent 
that the five categories identified as factors from the freelist exercise (organizational, curriculum, 
pedagogical, teaching techniques, student) were evident in respondents’ descriptions of their 
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planning steps; yet, in describing their actual behaviors, respondents discussed additional factors 
and strategies. New categories that emerged focused on actual strategies undertaken while 
planning as opposed to influential factors, including information retrieval (e.g., old lecture 
notes), information review (e.g., reviewing textbooks), artifact updating (e.g., revising old 
PowerPoint slides), artifact preparation, class preparation, and conducting post-class reviews. It 
was necessary in some cases to add nuance to the existing five categories as respondents 
discussed their behaviors at a finer level of detail. These included subdividing the category of 
teaching techniques into three parts: illustrations and examples designed to engage students and 
pique interest, problems or questions designed to elicit information, and the selection of 
instructional technology (see Table 3). 


Table 3. Most frequently reported planning steps by institution 



All Sample 


Institution 




Inst A 

Inst B 

Inst C 


n=47 

n=15 

n=14 

n=15 

Information retrieval: Notes 

23 

8 

7 

8 

Information retrieval: Slides 

14 

2 

4 

8 

Artifact updating: Slides 

14 

4 

5 

5 

Information review: Textbook 

12 

4 

3 

5 

Artifact updating: Notes 

12 

4 

2 

6 

Teaching technique: Problems or questions 

11 

5 

3 

3 

Curricular factor: Main topic of class 

10 

4 

3 

3 

Teaching technique: Illustrations or examples 

8 

4 

2 

2 

Artifact preparation: Write lecture 

7 

1 

5 

1 

Class preparation: Rehearse lecture 

7 

2 

3 

2 

Pedagogical considerations: How to best present material 

6 

2 

0 

4 

Curricular factor: Point in the syllabus 

6 

0 

4 

2 

Curricular factor: Link to previous class 

6 

1 

2 

3 

Teaching techniques: Instructional technology 

5 

2 

3 

0 


At this stage of planning curricular artifacts play a particularly influential role. In fact, the 
five most frequently reported planning steps directly involved the retrieval, updating, or review 
of a curricular artifact. While some degree of institutional and disciplinary variation is evident, 
more attention is paid to these differences in the next section that focuses on the artifacts 
themselves. This is because the source of the variation appears to be largely based on the nature 
of the artifact and not necessarily the planning step itself. Furthermore, some planning strategies 
such as retrieving premade artifacts seem to be “activated” across the entire study sample. This 
suggests that some planning strategies are commonly used regardless of how the institution or 
discipline influences the initial construction of the problem space (i.e., situation recognition) as 
well as the nature of the curricular artifacts themselves. In the following section, the five most 
frequently reported planning steps are briefly described. 

Information retrieval: Notes and slides. Many faculty in the study sample reported that 
in planning for a class they retrieved existing infonnation in the form of two types of curricular 
artifacts: lecture notes (23 respondents) and PowerPoint slides (14). In both cases, these artifact 
types had been created in previous years, and preparing for the current term involved retrieving 
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them from a file cabinet, desk drawer, or computer. As one geoscience instructor noted, all of her 
materials are “in the can” and just need to be retrieved annually. She taught the same course on a 
regular basis, so she viewed retrieval of the artifacts as both sensible in terms of time 
management and as a valuable resource that should be reused. 

Artifact updating: Notes and slides. Another frequently reported planning step is 
making updating PowerPoint slides (14) or lecture notes (12). Respondents used terms such as 
“fine tuning” or “tweaking” to describe this step. Updating included making topical changes 
based on new research, altering activities based on recollections about what worked or did not 
work in prior terms, and changing artifacts based on changes in the textbook or syllabus. For 
example, one geology faculty stated that, “For Wednesday’s lecture I will use part of the material 
that I had last year, but every year I tend to change and tune it as I can see that some concepts 
need more time than I expected; it’s mostly a rephrasing as I can see the students reacting to my 
lectures, so it’s more of a fine tuning.” 

However, this planning step only holds true if the instructor has taught the course before, 
or if he or she is new to a course but inherited materials from the previous instructor. In other 
instances where a new course must be created, a substantial amount of work will need to be 
invested in creating a syllabus, gathering materials, and so on. It is this investment of time and 
energy into curricular artifacts that made some respondents more inclined to engage in fine 
tuning each year rather than conduct a major revision or overhaul to their courses. 

Information review: Textbook. Respondents (12) also discussed reviewing information 
in another type of curricular artifact — the textbook — as an important planning step. In these 
cases, prior to the beginning of the term, or immediately before a specific class, the respondent 
read through the relevant chapters in order to refresh his or her familiarity with the course 
material. In most cases a text authored by another scholar was an important reference point 
during the planning process. However, in one instance a respondent had written a textbook in her 
field but did not reference it because it “was all in my head” and thus did not require consulting 
with any external curricular resource. 

Teaching technique: Problems or questions. Several faculty reported considering what 
types of problems or questions to include in a given class as one of their planning steps. This 
entailed identifying good problems to work out on the chalkboard, thinking of “thought- 
provoking questions” that would engage the class, or selecting questions for use in clicker- 
response systems. In this step, the instructor thought about and then identified problems or 
questions that illustrated the topic at hand while simultaneously leading students to engage with 
the material in a thoughtful and substantive manner. 

Curricular factor: Main topic of class. Ten respondents noted that the main topic of the 
class was an important part of their planning. That is, in response to the question about how they 
planned the course, they reported a step of simply thinking about the topic being taught. Thus, 
considering the course material itself became an important planning strategy. 
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3. Curricular Artifacts: What Is the Physical Form of the Plan for a Particular Class? 

As noted, the artifacts people use while planning are important because they shape 
activity by mediating the translation of intention into practice (Wertsch, 1991; Lave, 1988). 
Furthermore, the results reported thus far underscore the important role that curricular artifacts 
such as lecture notes play in the planning process, and so the next issue to probe is to what 
degree these artifacts are enacted in the classroom. The results indicate that plans for faculty in 
the study sample primarily take two physical formats: paper lecture notes (23 respondents) and 
PowerPoint slides (22). In addition, eight faculty reported using a combination of both lecture 
notes and slides (see Table 4). 

These results highlight the dominant use of these planning forms and indicate their 
entrenched use in academic settings. Given the importance of the enactment process whereby a 
static artifact is used to support or otherwise mediate action in the classroom (Remillard, 2005), 
it is useful to explore the three ways that respondents discussed artifact use: no role in their 
teaching, role in the classroom, and role outside of the classroom (Table 4). 


Table 4. Artifact types and planned enactment by discipline (n=47) 


All Sample 



Institution 


t"- 

II 

G 

Inst A 
n=15 

Inst B 
n=14 

Inst C 
n=15 

Artifact Type 





Paper notes 

23 

8 

8 

7 

PowerPoint slides 

22 

8 

7 

8 

Combination of note types 

8 

0 

3 

5 

Textbook 

4 

3 

1 

0 

Word documents 

3 

2 

0 

1 

Course manual 

2 

2 

0 

0 

No notes 

2 

0 

2 

0 

Pdf/tablet notes 

1 

1 

0 

0 

Planned enactment in class 





Has notes but no role in class 

3 

0 

3 

0 

In-class role 





Teaches from PowerPoint slides 

11 

2 

4 

5 

References while speaking 

7 

0 

5 

2 

Uses only for board work 

5 

3 

2 

0 

Uses notes with blanks 

3 

2 

0 

1 

Out-of-class role 





Posts online 

8 

5 

2 

1 

Rehearses before class 

7 

3 

1 

3 


First, three respondents noted that while they prepare lecture notes they do not bring them 
into the classroom. For these individuals, developing the curricular artifact was a way to get their 
thoughts on paper and to outline a sequence of topics or problems for an upcoming class, such 
that the purpose of creating the artifact (i.e., lecture notes in these cases) was preparatory and not 
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for use in the classroom. Second, 26 respondents discussed their use of artifacts (i.e., lecture 
notes and PowerPoint slides) in the classroom in various ways, including teaching directly from 
slides (11), using notes as a reference while speaking (7), for copying problems onto the board 
(5), or for using premade slides with blanks to fill in during class (3). In these cases, the notes or 
slides play a considerable role in guiding instruction because it is more or less directly 
“translated” from artifactual fonn to activity. Third, 1 5 respondents reported that their notes or 
slides play an important instructional role but not directly in the classroom. In these cases, eight 
faculty post their slides or notes on the course website prior to class and seven rehearse their 
lectures before the class period. Some respondents discussed using multiple strategies, so these 
data do not represent mutually exclusive approaches to artifact use. For example, one respondent 
prepares slides of calculus problems with blanks in certain areas, posts these online before the 
class, and during class engages students in collectively solving the problem. 

4. Two Cases of Curriculum and Instructional Practice: Two Tenure-track Physicists 
Teaching Medium-sized Upper Division Courses 

To examine how the processes of problem space construction action unfold in real-world 
settings, I examine in greater detail how two instructors planned and then taught a class. The 
results from these cases, as they map onto the model of problem space construction advanced in 
this paper, are depicted in Figure 3. 

Chiyoko. “Chiyoko” is an assistant physics professor at Institution A. The course 
discussed in the interview conducted for this study was on electrodynamics, and the syllabus was 
largely based on Griffith’s electrodynamics text. It was a junior level course, and Chiyoko 
observed that clickers were not necessary because the class wasn’t overly large, and no resources 
were available to do a more hands-on tutorial or demonstration-based class. When asked to list 
the factors that most influenced her course-planning activities, Chiyoko reported the following: 
time, course content, student struggles, application to practice, pedagogical goals, and staying on 
schedule. These factors thus represented the parameters of the course design problem space for 
this particular instructor. 

The impact of the first term (i.e., time), and the pressure that it put on her personal goal of 
ensuring that students were directly engaged with the material, became clear when she began to 
describe her approach to teaching. Chiyoko stated that she was “trying to do the poor man’s 
thing of how do you involve students without killing me on time and feeling like I am 
constrained into a lecture format.” She then elaborated on how time considerations, as well as the 
required course content, did not co-exist particularly well with her pedagogical goals, stating that 
basically she was “trying to get through the material in the class and staying on the syllabus 
schedule, while also trying to read where students are having trouble.” In these remarks about 
time constraints, she made clear that she was referring to both the limited time she had available 
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Step I: Situation Recognition Step 2: Planning steps 

Key factors influencing curriculum design Discrete steps taken to plan observed class 


Step 3: Artifacts and planned 
enactment 

Artifact developed for class and planned use 


Time 


Step 4: Actual enactment in the classroom 

Classroom observation results 



Respondent A I 8 


Consult w/staff 
for available 
demonstrations 


Artifact 

Notes 


Planned Enactment 

Hold notes while deriving at 
board 


Teaching techniques 
Lecturing (.04) 
Lecturing w/Slides (.00) 
Lecturing w/Board (.85) 
Small group work (.35) 

Pedag o gical strateg ies 
- Reads notes (.00) 
Illustrations (.08) 
Organization (.12) 
Assessment (.00) 




Class Topic 

Electrodynamics 


* 

T 


Look 


f Make ^ 





► 

/updates to 1 



L. n0t6S J 


Question ty pes 
Display questions (.42) 
Comprehension questions (.00) 

Cognitive engagement 
Creating (.00) 

Connections to real-world (.08) 

Instructional technology 
Notes (.42) 

PowerPoint (.00) 

Chalkboard (.85) 

Clickers (.00) 

Demonstration (.00) 




I . Dept Expectations 


2. Course 

3. Canon of 

History 

Field 


• 

4. Nat’l 

jy 5. Student 

Expecations 

Struggles 



Rehearse 

lecture 


Artifact 

PowerPoint slides; notes 

Planned Enactment 

Post notes online; use 
PowerPoint slides in class (and 
post them too) 


6. Student Interests 
7. Past experience w/course a 


Class Topic 

Damped-down harmonic motion 


Teaching techniques 
Lecturing (.15) 
Lecturing w/Slides (.41) 
Lecturing w/Board (.31) 
Small group work (.44) 

Pedag o gical strateg ies 
Reads notes (.00) 
Illustrations (.08) 
Organization (.03) 
Assessment (.21) 


Respondent C I 6 


Question ty pes 
Display questions (.77) 
Comprehension questions (.08) 

Co gnitive eng a gement 
Creating (.00) 

Connections to real-world (.08) 

Instructional technolog y 
Notes (.00) 

PowerPoint (.69) 

Chalkboard (.33) 

Clickers (.41) 

Demonstration (.08) 


c 


Make notes after class 


3 


Figure 3. Two instructors mapped onto a problem space model 


20 





Navigating the Problem Space of Academia 


in her job writ large, as well as the time constraints within a 50-minute class period. Then, her 
focus on identifying how students were doing in the course was grounded in her attempt to get 
students to “conceptualize problems beyond the math,” especially for the large group of students 
who were neither high achieving or interested. With the problem space of planning and teaching 
thus established in terms of time constraints and considerations about the content and student 
learning, Chiyoko then described how she went about planning for a specific class. 

The first step she described in planning for the observed class, which was on 
electrodynamics, was to consult with the person who coordinates demonstration equipment in 
her department to see what equipment was available. Then Chiyoko looked at her old notes and 
updated them based on new developments in the research literature or the news that may be of 
interest to students. Thus, her planning steps for the observed class centered on consulting with 
departmental support staff, retrieving old notes, and fine tuning them. 

This process culminated with a revised version of her lecture notes. These notes played 
an important role in the classroom, especially as she planned to work through a series of 
derivations in the observed class. As Chiyoko noted, “I’m not good at deriving without notes for 
the board, so I’ll actually hold my notes in my hand as I go through and put it on the board.” In 
the observed class, she spent most of the instructional time lecturing at the chalkboard (behavior 
observed in 85% of all 2-minute intervals), but also organizing students in small groups to 
discuss their work (35%). She also asked many questions seeking original answers (42%) as 
opposed to rhetorical questions, and directly referenced her notes frequently (42%). 

In summary, Chiyoko’s conceptualization of the “problem” of planning and then teaching 
a class on electrodynamics was largely framed by concerns about time (both within the class 
period itself but also in her workload more generally), and covering the material in ways that 
satisfied both the pressures of the course syllabus and the needs of her students. The strategies 
that she then engaged in to plan the class included retrieving and fine tuning lecture notes, which 
in turn played an instrumental role in the classroom. Indeed, the key role that artifacts play in the 
planning process is evident in this case, with little to no translation between the artifact (i.e., 
lecture notes) and practice. Further, her pre-existing goals regarding student engagement were 
evident in the classroom through her use of small group work and regularly asking questions. In 
these ways, her teaching practices appear to be shaped by a combination of personal goals, 
situational constraints, and the curricular tools at hand. 

Gary. “Gary” is a full professor who has worked at Institution C since 1993, where he 
has long been deeply engaged in physics education research. His situational awareness included 
references to departmental expectations, history or what was in the course before, canonical texts 
or what he called “universally known knowledge,” national expectations for physics education, 
student struggles and interests, and his past experience with the course. For Gary, concerns about 
the canon are the starting point for thinking about the curriculum in general and a specific class 
on harmonic motion. In addition to the other factors noted in the freelist exercise, he emphasized 
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that his past experiences with the course play an important role, and that “you can’t help but have 
that influence” how the course is planned and then taught. 

For the class being observed, which was focused on the topic of damped simple hannonic 
motion, Gary drew upon materials he developed the prior year. He taught the course a year before 
in collaboration with a postdoctoral student who was part of the teaching-refonn effort at Institution 
B. The purpose of that intervention was to provide a postdoc for 2 years to work on a specific class 
to develop new curricular artifacts (e.g., PowerPoint slides, clicker questions, in-class activities) 
based on educational research in physics. A primary reason for focusing on creating high-quality 
artifacts was that the departmental policy for course rotations, which involved new instructors 
taking over several undergraduate courses every few years, essentially led to the wholesale 
recreation of the course based on the incoming teachers experience and views. Gary felt that by 
creating a “binder” of lecture notes, clicker questions, and detailed information about recurrent 
student misconceptions, the next instructor to take over the course would be more inclined to adopt 
(and adapt) these materials rather than creating entirely new ones whose quality would be unknown. 

When mentioning “departmental expectations” as a key factor influencing his planning, 
he was thinking about the expectations for interactive teaching held by the local (and national) 
community of researchers involved in this pedagogical reform. Additionally, Gary had long been 
involved in education research in his field, based on a strong commitment to improving his own 
teaching as well as to his students. Thus, the problem space for Gary’s planning and teaching 
largely consisted of considerations about his students’ learning, the history and content of this 
particular course, and his colleagues’ expectations of what quality teaching looks like. 

The plan for the class was to lecture for approximately half of the time, and to use 
interactive tutorials for the remainder of the period. The specific materials used for this class were 
inherited from the previous instructor, and included clicker questions, a course schedule, and 
lecture notes. In preparation, Gary reported that he “looked at my old lecture notes and PowerPoint 
slides, rewrote and reordered the slides, and stood at the blackboard (in my office) and went 
through the lecture portion.” He also has handwritten notes from the previous year for each class 
and usually revises them, but for the class being observed the notes were mostly in his PowerPoint 
slides and “in my head.” Gary also posted pdf files of his notes and slides on the course website. 

During the observed class, Gary used a mixture of lecturing approaches (e.g., with 
PowerPoints observed in 41% of all 2-minute intervals, board work in 3 1%, and with no media 
in 15%) and small-group work (44%). He also used a considerable amount of assessments (21%) 
that were often carried out through verbal questions seeking original information (77%) and 
clicker technology (41%). These teaching behaviors were largely consistent with his self- 
reported plan, where the class would be half lecture and half tutorials. At no point was Gary 
observed using his notes, but his slides did serve an important organizational function by pacing 
the class and providing opportunities for questioning. Overall, the planning and teaching 
practices of Gary reflect a consistent thread from original recognition of the situation all the way 
through to classroom instruction. That is, his situational awareness of student learning of the 
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course material and related expectations of the education community acted as an overarching 
frame for how he approached the class. Then, because materials had already been prepared for 
the class based on strong considerations of student learning, planning involved retrieving, fine 
tuning, and then rehearsing these curricular artifacts. In this way, in retrieving these artifacts his 
actual planning practices were consistent with the rest of the study sample, but the problem space 
itself was framed more by student-related factors than by time constraints and his workload. 

Discussion 

In this study I set out to describe how faculty plan and then teach their courses in real- 
world settings. The results indicate that a core set of fixed organizational and curricular factors 
are widely perceived as influential contextual affordances, with workload and attendant time 
limitations being particularly important in contributing to the routine maintenance of pre-existing 
artifacts (i.e., lecture notes and PowerPoint slides). In this section I elaborate on key aspects of 
these results, and discuss implications for current policy and programmatic efforts that encourage 
faculty to adopt interactive teaching techniques. 

A New Model for Faculty Practice: The Problem Space of Curriculum and Instruction 

Much like earlier work on curriculum design in higher education, which sought to 
identify all variables that influence the planning process (e.g., Conrad & Pratt, 1983; Lattuca & 
Stark, 2011), I too aimed to find those critical personal and contextual factors that appear to 
shape, if not dictate, how faculty plan and then teach their classes. However, this work differs 
from these earlier attempts in shifting the primary analytic focus from the omniscient observer 
who derives comprehensive models of the world, to that of the teachers’ own perspective and 
experience. That is, the emphasis is placed on the actual views, perceptions, and experiences of 
individuals on the ground, or what anthropologists call the emic perspective. 

This shift is informed by three developments. First, the critique that to account for all 
possible influences operating in real-world situations (both consciously perceived and tacitly 
influential) is nearly impossible and raises significant methodological concerns should not be 
ignored (Lee & Porter, 1990). Second, the evidence is clear that teachers make instructional 
decisions largely based on their own pre-existing cognitive frameworks that filter new infonnation 
(e.g., new policies governing teaching), frame situations, and guide action (Shavelson & Stern, 
1981; Fives & Buehl, 2013). Further complicating matters is the third development, namely that 
these cognitive frameworks are themselves influenced by the contexts in which faculty work, such 
that a wide range of factors (e.g., organizational, sociocultural, and personal) dynamically interact 
to shape instructional decision making (Greeno, 1998). This should dissuade researchers and 
policymakers from continuing to seek that single variable, or magic bullet, that will both single- 
handedly explain behavior as well as transform undergraduate education (Umbach, 2007). 
Instead, viewing the world through the eyes of instructors on the ground, and the more limited 
but influential combinations of variables influencing them, may be more productive in seeking to 
understand the dynamics of educational reform (Spillane, Reimer, & Reiser, 2002). 
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The application of problem space theory as outlined in this paper results in such a 
grounded, ernic account of faculty practice that builds upon these empirical and methodological 
developments in the field of educational research. In brief, the model outlines the processes 
whereby faculty construct the problem space of course planning, which is initiated first by 
actively perceiving a limited number of factors or conditions that are considered particularly 
salient to their work (e.g., class size). Once perceived, a variety of strategies or action steps occur 
to the problem-solver in relation to these factors in a process of successively narrower options 
(e.g., no small group work for large classes) that ultimately leads to the development of a 
curricular plan and then enactment in the classroom. I suggest that this model opens up the black 
box of instructional decision making, particularly the subtle dynamics among actor, situation, 
and artifact that comprise faculty practice “in the wild” (Halverson & Clifford, 2006). Insights 
into these complex phenomena can be used to improve educational leaders’ understanding of 
faculty work and, consequently, the design and ultimate efficacy of teaching-related interventions. 

Nonetheless, this work is in its early stages of conceptual and methodological 
development, and several limitations are immediately apparent. Perhaps most pressing, given the 
importance of cultural forces in academia that govern norms for teaching and the nature of 
knowledge itself (Clark, 1983; Tierney, 1988), is the prospect that too much explanatory power 
may be placed on individual agency and conscious sense-making processes, which serves to 
obscure the influence of the cultural context and its subtle or tacit influence on individual and 
group action. A theoretical stance that foregrounds the impact of the cultural context on behavior 
is provided by field theory, which is often explained in terms of a game, where the field of 
activity is governed by both explicit and tacit rules that all players recognize and follow when 
making decisions (Martin, 2003; Winkle- Wagner, 2012). Thus, behavior is largely based on how 
an individual perceives a particular culturally organized field of play, and which of its delimited 
number of subsequent options are most optimal in a given situation. The intersection between 
field theory and a more cognitivist approach as outlined in this paper represents a potentially 
productive area of future research that could elaborate the role of culture in instructional decision 
making in a theoretically robust way. 

In the remainder of this section I focus on three key elements of instructional decision 
making that seem particularly influential in shaping how faculty think about and then actually 
plan and teach their courses. Then, I explore the relationships among these decisions, curricular 
artifacts, and classroom teaching, followed by implications of this study for policy and practice. 

The Dominant Features Perceived to Shape the Problem of Planning 

When they considered the factors most salient to their course planning activities, faculty 
in the study sample most frequently reported a set of factors that could be considered “fixed,” or 
somewhat non-negotiable features of their organization’s operations and policies. These factors 
included time, course content, and class size, each of which are linked to workload obligations, 
course curricula, and departmental policies governing course size, respectively. As such, the 
problem space for planning is demarcated less by idiosyncratic or context-free considerations 
such as personality or pedagogical beliefs, and more by an awareness or attunement to what is 
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demanded by and possible within the existing organizational context. This is not surprising, 
given evidence that teachers’ generally view their adoption of new policies or innovations, such 
as interactive teaching techniques, through the lens of what is feasible and desirable within their 
current workplace (Coburn, 2001; Henderson & Dancy, 2007). 

However, the results also suggest that the role played by fixed organizational affordances 
in how faculty construct the problem space for teaching is not dominant across all institutions or 
disciplinary groups included in this study. For example, at Institution B the most salient factors 
were teaching techniques (i.e., examples and illustrations) and student factors (i.e., student career 
or degree trajectory). Further, disciplinary variations were observed. The most salient factor for 
physicists was departmental expectations, for biologists pedagogical goals were critical, and for 
chemistry faculty the background of students was reported as the most salient factor. Thus, it 
appears that the factors perceived as salient vary as one gets into smaller organizational units, 
each of which operates within its unique institutional and sociocultural context. 

These results differ from research on this topic conducted more than 20 years ago, where 
Stark (2000) found that the contextual factors viewed as most influential in regard to course 
planning included student-based topics such as student characteristics (e.g., ability and level of 
preparation) and goals. These considerations of students were followed by what Stark (2000) 
called “pragmatic issues” that included class size, followed by factors such as influences external 
to the college (e.g., accreditation) and college goals. In contrast, faculty in the current study 
exhibited a relatively low saliency for student-based factors as one of the key elements 
considered while planning. With the exception of Institution B, faculty report the salient features 
of the problem space for course planning as being exclusively comprised of non-student factors. 

Some interesting questions arise from this finding. What are the implications of a 
planning process that omits a deep consideration of student needs and abilities? Is it preferable to 
have faculty think about their students when planning a course, and, if so, why? In the case of 
Chiyoko, her personal goal to engage the “middle” group of students who were neither high nor 
low achievers, along with a focus on student struggles with particular topics, certainly appeared 
to have shaped how she designed her classes and selected activities in a way that led to an active 
and engaging classroom. At the very least, it seems that some consideration of students during 
the planning process, as is evident in approaches such as backwards design or problem-based 
learning, contributes to instruction that is well aligned with the interests and aptitudes of a 
particular group of students. The relative absence of student-based factors in the study sample 
may be indicative of a population for whom other factors, particularly that of time and related 
workload constraints, enters into their thought process more readily and easily. 

The Centrality of Time Constraints and Implications for Routine Maintenance 

A major finding from this study pertains to the centrality of time constraints on how 
faculty perceive their own work and the concomitant possibilities (or lack thereof) for 
instructional improvement. To a person the faculty participants in this study were extremely busy 
professionals with many competing demands on their time such as running labs, mentoring 
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students, and teaching both graduate and undergraduate courses. Even for those not on the tenure 
track for research-intensive positions, many lecturers in the study were involved in educational 
research activities or otherwise engaged in research-related pursuits. All told, for this population, 
time was a particularly valuable and scarce resource. 

As perceived affordance theory suggests (Greeno, 1994), over time teachers will become 
attuned (and even habituated) to desirable and feasible courses of action related to particular cues 
or situations, and which cues in the environment should be given more or less attention. So how 
does this perceived affordance influence practice? First, it contributes to the reliance on a 
planning and class preparation strategy that involves the least amount of time — that of fine 
tuning or tweaking existing materials, a strategy that Stark (2000, p. 420) called “routine 
maintenance.” Maintenance often involves improving problem sets, integrating new teaching 
strategies or activities, or updating the course material in light of new scientific discoveries. This 
entrenched strategy is altered only in certain circumstances, such as when a course is brand new 
and requires an entirely new curriculum, instructors are strongly dissatisfied with a course, or 
when a new instructor comes into a course. These results are consistent with Stark’s findings 
from more than 20 years ago, which suggest that the routine maintenance of curricular artifacts is 
a deeply entrenched cultural practice among postsecondary faculty in research universities. 

What are the implications for this reliance on line tuning existing materials? The answer 
to this question depends in part on the quality of the materials and whether or not they are 
designed with attention to not only the rigorous presentation of content but also to engaging 
presentation, the scaffolding of material from less to more complex, and other principles of 
effective teaching. If poorly conceived syllabi or activities are created, stored, and annually 
retrieved, improvement to the teaching of the person using these materials is difficult to envision. 
Conversely, if the materials are of high-quality, even if a time-pressed instructor simply takes 
them out of the drawer an hour before class, it is possible the class will be more pedagogically 
informed than otherwise. A key to this process seems to be ensuring that the artifacts are well 
done, but therein lies the catch-22 for faculty, as preparing such materials requires a resource that 
is in short supply — time. This quandary was highlighted by a biology instructor who was a 
fulltime lecturer and deeply involved in biology education, and yet: 

I would prefer to start every class period or every topic with an active exploration on [the 
student’s] part, culminating with some wrap up by me. But in practice there are some topics 
that I don’t start that way. And it’s just because I haven’t developed the right sequence of 
things that I think will trigger them to really learn it that way. And so it’s just time really on 
my part, [whether or not] I had time to develop some [materials] I think will actually get 
them going. And if I haven’t had time to address that concept [or] to create some cool 
activity for them to do then I revert back to the old fashioned way of introducing the 
concept and then maybe writing some clicker questions that get them to think about it. 

So for this instructor, who has the training, motivation, and position to implement 
interactive teaching as advocated by policymakers and educators, to actually prepare quality 
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materials requires a substantial amount of time. Thus, even for faculty who are deeply committed 
to improving their teaching, within the parameters of their work and careers the routine 
maintenance of existing artifacts is simply a sensible and in some cases the only strategy. 

In addition, if the workload and attendant time constraints are the primary lens through 
which teaching activities are viewed, there may also be implications for educational refonn given 
evidence that new policies and initiatives are also “filtered” or interpreted through the cognitive 
frameworks of teachers on the ground. Thus, if the workload is sufficiently heavy to not 
reasonably allow for or accommodate the demands of a new policy (e.g., introducing clickers 
that require time for training and incorporation into the curriculum) it is unlikely to succeed 
unless something is done to alter the underlying organizational conditions and their 
corresponding perception of constraints and affordances for practice. 

Does the Artifact Matter? Enactment in the Classroom 

The data indicate that curricular artifacts mediated the interactions between teachers and 
students in two important ways: as a guide to classroom instruction or as a study aid where notes 
or slides were provided directly to students. Once created and/or retrieved, the faculty enacted 
the slides and lecture notes in the classroom with varying degrees of fidelity to the original. For 
some, the notes were simply guides referenced every few moments but did not play a central role 
in teaching. For others, the notes included entire sentences or solutions to problems that were 
either recited verbatim or copied onto the chalkboard in their entirety. In both cases, the notes or 
slides acted as a reference point during instruction to ensure that the correct solutions and/or 
facts were conveyed and also to keep the instructor on schedule, which suggests that the form 
and content of the artifacts influences classroom teaching. 

However, some situations or aspects of teaching may minimize the influence of artifacts. 
In some cases faculty reported walking into the classroom “cold” with no notes, materials, or 
evident preparation, which renders curricular artifacts unimportant and makes the instructor’s 
mental acuity the primary resource. For example, the geologist who wrote the primary textbook 
in the field said that everything was “in my head” and her classes on a given day were guided by 
a combination of the syllabus and her mood and inspiration. Yet, this approach to teaching was 
not common in the study sample; even experienced instructors emphasized the importance of 
well-designed artifacts and preparation in order to ensure their peace of mind as well as the 
quality of instruction. Additionally, several faculty emphasized the performative aspects of 
teaching, such that even in cases where multiple classes were taught using the same set of 
artifacts the classes could substantially differ in tone and composition depending on the student 
body or the instructors’ mood. This is especially the case in regard to the different types of 
questions students may pose, and the resulting conversations or discussions that may ensue. 

Thus, the human dynamic and subtle features of communications between teachers and students 
should not be ignored when considering how artifacts mediate classroom teaching. 
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Implications for Policy and Practice 

Besides contributing new empirical insights into instructional decision making and 
curricular design in postsecondary settings, the results reported in this paper have implications 
for educational practice and change. Before proceeding with this discussion, it is important to 
revisit the fact that multiple factors shape how teachers plan and teach their courses such that no 
magic bullet or single factor (e.g., tenure and promotion criteria) can be altered to transform or 
predict practice. This fact alone should dissuade the field from continuing to seek that single 
variable or policy that will transform undergraduate education. Instead, much is gained in 
obtaining a realistic and multifaceted understanding of how faculty think and act in their 
workplaces, an understanding that can be used as an instructional guide of sorts that practitioners 
and policymakers alike can use to design new programs that are more aligned with local practice 
than not (Umbach, 2007; Helman, 1997; Rogers, 1995). The framework outlined in this paper 
provides a conceptual lens that, if used to diagnose or study how faculty navigate their 
organizations when planning and teaching their courses, could reveal particularly propitious 
variables or “levers of change” that, if altered or supported, may result in changes to practice 
(Spillane, Halverson, & Diamond, 2001; Cobb, Zhao, & Dean, 2009). 

The main result reported in this paper that can reasonably be thought of as a potential 
leverage point for change is that of time constraint and its relationship to the strategy of routinely 
fine tuning existing notes or slides. Given that the academic workload is unlikely to change in 
the near future such that more time is available for course preparation, and even may get more 
demanding in tenns of responsibilities and obligations expected of both tenure-track and 
contingent faculty, I suggest that in seeking solutions to the potentially short-sighted habit of 
routine maintenance of notes and slides it is instead more tenable and feasible to ask faculty to 
do one simple thing, namely to engage in a practice described by three different instructors in 
this study: take notes immediately after class about what worked and what did not work in terms 
of teaching style, activities, sequencing of material, and so on. Upon describing this practice and 
being asked if she wrote these notes in her office after the class, one biologist stated that: 

No, I do not have that much time. I can tell right during the lecture that things are not 

working, and in the few minutes that I am packing up after lecture, I will make a note 

right on the printout of the PowerPoint slides. 

This respondent then goes on to state that this was an effective practice only because she 
was well organized and has a notebook containing all materials required for teaching the course, 
including the syllabus, attendance records, notes about individual students, and printouts of 
slides for each lecture. Thus, when notes are made on the slide printout, when she teaches the 
course the next semester she will immediately see and reflect upon the new observation and 
incorporate that into the next iteration of the class. 

While this brief and even cursory reflection about their teaching is not exactly what is 
recommended by those who advocate reflective practice as a cornerstone to improving 
professional practice in general or teaching in particular (e.g., Schon, 1983; Jay & Johnson, 
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2002; Kane, Sandretto, & Heath, 2004), given the fixed constraints perceived by today’s faculty 
members in regards to their workload and attendant priorities, it is unfortunately not likely that 
postsecondary teachers will be able to allocate hours to keeping a teaching journal or consulting 
with a faculty developer. Instead, asking or even requiring faculty to make brief observations of 
how each class went would represent an important step in developing a sense of reflection and, 
ideally, continuous improvement in their practice. 

The prospect of making such reflection on the success or failure of the course a 
departmental policy was actually raised by a respondent, who observed that his departmental 
chair did not require a post-course write up about how the semester went, but that “maybe he 
really should.” Ideally, these reflections would directly lead to updating the instructor’s lecture 
notes and/or PowerPoint slides for the following year or semester, in which case the benefits of 
the brief period of reflection would be twofold: improved curricular artifacts and an instructor 
who is beginning to become a reflective practitioner. Would this be asking too much of already 
over-burdened faculty to respond to yet another administrative task? Perhaps, but the benefits of 
engaging in reflective practice outweigh the potential backlash that departmental leaders may 
receive. In requiring faculty to think and write about their teaching efficacy, their notion of the 
problem space for teaching could potentially be extended beyond the immediate and habituated 
factors of time and content to encompass additional considerations of pedagogy and student needs. 

Conclusions 

The traditional manner of teaching and conveying knowledge and skills to the next 
generation is changing in the face of digital technologies and prodigious efforts to encourage 
faculty to become less of a “sage on the stage” and more of an engaged mentor with a diverse 
pedagogical toolkit. Such developments raise the prospects that the multifaceted factors that 
faculty must negotiate in their work will only increase, and attempts to both understand and 
change how they plan and teach courses will become correspondingly more complex. Adding to 
the challenging nature of attempting to discern the “causes” or “determinants” of behavior is the 
fact that much of social action is shaped by tacit or subconscious compliance with the unspoken 
rules of the game (Martin, 2003; Greeno, 1998). Thus, any attempts to model the key features 
that shape faculty sense making and subsequent strategies for their teaching-related work must 
proceed cautiously and with careful attention to the limitations of such an approach and the 
theoretical insights offered by other lines of inquiry pursuing similar topics, such as naturalistic 
decision making (Klein, 2008), figured worlds theory (Holland, Lachicotte, Skinner, & Cain, 
1998), and field theory (Martin, 2003). Ultimately, researchers and policymakers engaged in 
educational improvement at the postsecondary level will need to recognize that individual 
faculty — and the variety of cultural spaces they inhabit — cannot be managed or controlled, 
which was and remains a dominant way of thinking in regard to organizational improvement 
(e.g., Martin, 2002). Instead, to advance the quality of undergraduate education within the 
existing structures and cultural systems at play in colleges and universities across the United 
States, policymakers and practitioners must strive to find that critical balance between “best 
practices” and the lived experiences of faculty on the ground. 
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